
REMARKS 

By this Amendment, claims 1-16 are amended. Thus, claims 1-16 are active in the 
application. Reexamination and reconsideration of the application are respectfully requested. 

The specification and abstract have been carefully reviewed and revised to correct 
grammatical and idiomatic errors in order to aid the Examiner in further consideration of the 
application and to correct the informalities identified in item 1 on pages 2-3 of the Office Action. 
Further, the abstract was revised in order to delete numerical references to the drawings. The 
amendments to the specification and abstract are incorporated in the attached substitute 
specification and abstract. No new matter has been added. 

Also attached hereto is a marked-up version of the substitute specification and abstract 
illustrating the changes made to the original specification and abstract. 

The Applicants thank the Examiner for kindly that the application is in condition for 
allowance except for the informalities of the specification identified in item 1 on pages 2-3 of the 
Office Action. Minor editorial revisions were made to claims 1-16 in order to improve their U.S. 
form. The present amendment of claims 1-16 was not submitted earlier since the necessity of the 
amendments was not noticed earlier and since the Ex Parte Quayle Action was issued as the first 
Office Action on the merits. Therefore, the Applicants respectfully request entry of the present 
amendments to claims 1-16. Further, the Applicants submit that the revisions made to claims 1- 
16 were not to broaden or narrow the scope of protection for the present invention. Accordingly, 
the Applicants respectfully submit that claims 1-16, as amended, are in condition for allowance for 
the reasons identified in item 3 on pages 3-4 of the Office Action. 

In view of the foregoing amendments and remarks, it is respectfully submitted that the 
present application is clearly in condition for allowance. An early notice thereof is respectfully 
solicited. 
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If, after reviewing this Amendment, the Examiner feels there are any issues remaining 
which must be resolved before the application can be passed to issue, it is respectfully requested 
that the Examiner contact the undersigned by telephone in order to resolve such issues. 



JRB/pth 

Washington, D.C. 20006-1021 
Telephone (202) 721-8200 
Facsimile (202) 721-8250 
May 28, 2004 



Respectfully submitted, 



Nobuhiko YASUI et al. 




Jonathan R. Bowser 
Registration No. 54,574 
Attorney for Applicants 
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Field of the Invention 



[0001] 



The present invention relates to rendering devices and, 



more specifically, to a rendering device which can be incorporated 
in a drive assistant device . In more detail , the rendering device 
generates a display image of an area around a vehicle based on 
10 an image that is captured by an image capture device fixedly placed 
in the vehicle. 

Description of the Background Art 
[0002] The drive assistant device incorporating such a 

15 rendering device as described above has been actively researched 
and developed. A conventional-type drive assistant device is 
mounted in a vehicle, and generally includes an image capture device, 
a rudder angle sensor, a computing unit, a rendering device, and 
a display device. The image capture device is fixedly placed in 

20 a predetermined position in the vehicle, and takes charge of the 
image capture device is provided for capturing an image of an area 
that is defined by its own the viewing angle of the image capture 
device . The resulting image is now hereinafter referred to as 
a captured image. The rudder angle sensor is also fixed in a 

25 predetermined position in the vehicle, and detects to what degree 

1 




the steering wheel of the vehicle is turned . Based on the detection 
result, the computing unit calculates a path an estimated path 
for the vehicle to take. The rendering device then renders the 
estimated path on the captured image, and the image generated 
5 thereby is ouch a display image such as the one shown in FIG. 
20. The display image is displayed on the display device. 
[0003] With such a_display image on the display device, a driver 
of the vehicle can know if his/her current steering will fit the 
vehicle in the a parking space without colliding into any obstacle 

10 in a close range of the drivers vehicle . If his/her steering 
is not appropriate, the estimated path is displayed out of the 
parking space in the display image. Therefore, the driver can 
appropriately adjust the rudder angle of the steering wheel. 
[0004] There is another type of conventional drive assistant 

15 device exemplarily disclosed in Japanese Patent examined 
Publication No . 2-36417 (1990-36417) . The drive assistant device 
additionally carries an active sensor for measuring a distance 
between the vehicle and an obstacle that is observed near the 
estimated path. Based on the measurement result provided by the 

20 active sensor, the computing unit determines which part of the 
estimated path is to be rendered on the captured image. The part 
thus which is determined to be rendered on the captured image 
is now hereinafter referred to as a rendering estimated path. 
In this manner, the rendering device accordingly renders on the 

25 captured image the rendering estimated path, which ends right 

2 




before the obstacle. 

[0005] The above — above-described conventional drive 
assistant devices carry have the following twoproblems as f ollowo 
First, the estimated path is fixedly determined in color for display 
5 Thus, even if the color is similar in tone to a predominant color 
of the display image, the color is unchangeable. Here, the 
predominant color is mainly determined by the road, for example, 
regardless of whether the road paved or not with asphalt . If this 
is the case, the driver finds it difficult to instantaneously locate 

10 the estimated path on the display image. 

[0006] Second, the estimated path that is rendered in the 
display image is represented simply by lines , failing which fails 
to help the driver instantaneously perceive how far he/she can 
move the vehicle . More specif ically, as shown in FIG . 21, a vehicle 

15 Vusr carrying the conventional drive assistant device is moving 
toward an obstacle Vbst. In this case, the vehicle Vusr first 
collides into a corner point Pcnr of the obstacle Vbst f not 
intersection points Pcrg of an estimated path Pp and the surface 
of the obstacle Vbst. ^fe— This m eans that the farthest point 

20 possible for the vehicle Vusr to move is the corner point Pcnr 
of the obstacle Vbst. As such, even if the estimated path is so 
rendered as to end immediately before the object, the second problem 
remains yet unsolved. 

25 
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SUMMARY OF THE INVENTION 

[0007] Therefore, an object of the present invention is to 
provide a rendering device, a device which generates display image 
being — generated — thereby that shows an estimated path in an 
5 eye-catching manner for the driver to easily locate. 

Another object of the present invention is to provide 
a rendering device, — a— device which generates display image 
generated thereby being that is indicative and helpful for the 
driver to know how far he/she can move the vehicle. 
10 [0008] The present invention has the following features to 
attain the above-described objects above . 

[0009] A first aspect of the present invention is directed to 
a rendering device for generating a display image of an area around 
a vehicle for drive assistance. The rendering device comprises 

15 a reception part for receiving a current rudder angle of a steering 
wheel of the vehicle from a rudder angle sensor fixed therein in 
the vehicle ; a derivation part for deriving an estimated path for 
the vehicle to take based on the rudder angle received by the 
reception part; and an image generation part for generating the 

20 display image based on a captured image which is captured by an 
image capture device fixed in the vehicle, and the estimated path 
that is derived by the derivation part . Here, in the display image, 
the estimated path is overlaid on an intermittent basis. 
[0010] A second aspect of the present invention is directed 

25 to a rendering device for generating a display image of an area 
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around a vehicle for drive assistance. The rendering device 
comprises a first reception part for receiving a distance to an 
obstacle that is located around the vehicle from a measuring sensor 
placed in the vehicle; a first derivation part for deriving a 
5 farthest point for the vehicle to move based on the distance received 
by the first reception part; a second reception part for receiving 
a current rudder angle of a steering wheel of the vehicle from 
a rudder angle sensor fixed in the vehicle; a second derivation 
part for deriving an estimated path for the vehicle to take based 

10 on the rudder angle received by the second reception part; and 
an image generation part for generating the display image based 
on a captured image which is captured by an image capture device 
fixed in the vehicle, the farthest point derived by the first 
derivation part, and the estimated path derived by the second 

15 derivation part. 

[0011] These and other objects, features, aspects and 
advantages of the present invention will become more apparent from 
the following detailed description of the present invention when 
taken in conjunction with the accompanying drawings. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0012] FIG. 1 is a block diagram showing the hardware structure 
of a rendering device Urndl according to a first embodiment of 
the present invention; 
25 FIG . 2 is a diagram showing a display image Sout generated 
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by a processor 1 of FIG. 1; 

FIG. 3 is a diagram showing a position where an image 
capture device 4 of FIG. 1 is placed; 

FIG. 4 is a diagram showing a captured image Sept captured 
5 by the image capture device 4 of FIG. 1; 

FIG. 5 is a flowchart showing the processing procedure 
of the processor 1 of FIG. 1; 

[0013] FIG. 6 is a diagram showing a left-side trajectory Ppl 
and a right-side trajectory Pp2 derived in step S6 in FIG. 5; 
10 FIG. 7 is a diagram showing overlaying position data 

Dsp generated in step S7 in FIG. 5; 

FIG. 8 is a diagram showing the display image Sout 
generated in step S8 in FIG. 5; 

FIG. 9 is a diagram showing the display image Sout 
15 generated in step S15 in FIG. 5; 

FIG . 10 is a block diagram showing the hardware structure 
of a rendering device Urnd2 according to a second embodiment of 
the present invention; 

[0014] FIG. 11 is a diagram showing a display image Sout 
20 generated by a processor 21 of FIG. 10; 

FIG. 12 is a flowchart showing the processing procedure 
of the processor 21 of FIG. 10; 

FIG . 13 is a block diagram showing the hardware structure 
of a rendering device Urnd3 according to a third embodiment of 
25 the present invention; 




FIG. 14 is a diagram showing a display image Sout 
generated by a processor 41 of FIG. 13; 

FIGS. ISA and 15B are diagrams showing placement 
positions of active sensors 441 to 444 of FIG. 13; 
[0015] FIG. 16 is a flowchart showing the processing procedure 
of the processor 41 of FIG. 13; 

FIG. 17 is a diagram for demonstrating the process in 
step S43 in FIG. 16; 

FIG. 18 is a diagram for demonstrating the process in 
step S44 in FIG. 16; 

FIG . 19 is a detailed diagram showing an estimated region 
Rpt generated in step S410 in FIG. 16; 

FIG. 20 is a diagram showing a display image displayed 
by a conventional drive assistant device; and 

FIG. 21 is a diagram for explaining problems unsolvable 
by the conventional drive assistant device. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS DETAILED DESCRIPTION OF 
THE INVENTION 

20 [0016] FIG. 1 is a block diagram showing the hardware structure 
of a rendering device Urndl according to a first embodiment of 
the present invention. In FIG. 1, the rendering device Urndl 
includes a processor 1, a program memory 2, and a working area 
3. The program memory 2 is typified by ROM (Read Only Memory), 

25 and stores a program PGa for defining the processing procedure 
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in the processor 1. By following the program PGa, the processor 
1 generates such a display image such as the display image Sout 
as shown in FIG. 2. The display image Sout shows a— an estimated 
path Pp ootimatcd for a vehicle Vusr {see FIG. 3) to take in the 
5 course of time. The estimated path Pp is composed of a left-side 
trajectory Ppl and a right-side trajectory Pp2 indicated by / 
respectively, which are indicated by indicators Sindl and Sind2 1 _ 
respectively . Here, the left-side trajectory Ppl is for a 
left-rear wheel of the vehicle Vusr, while the right-side 
10 trajectory Pp2 by — is for a right-rear wheel of the vehicle . 
Further, the indicators Sindl and Sind2 are both objects in a 
predetermined shape (e.g., circle, rectangle) that is previously 
stored in the program memory 2 . 

[0017] The working area 3 is typified by RAM {Random Access 
15 Memory) , and used when the processor 1 executes the program PGa. 
The rendering device Urndl 4rPt — according to the above 
above-described structure is typically incorporated in a drive 
assistant device Uastl. The drive assistant device Uastl is 
mounted in the vehicle Vusr, and includes at least one image capture 
20 device 4, a rudder angle sensor 5, and a display device 6 together 
with the rendering device Urndl. 

[0018] As shown in FIG. 3, the image capture device 4 is embedded 
in the rear-end of the vehicle Vusr, and captures an image covering 
an area to the rear of the vehicle Vusr. The resulting image is 
25 a captured image Sept as shown in FIG. 4. The rudder angle sensor 




5 detects a rudder angle 6 of the steering wheel of the vehicle 
Vuor Vusr , and transmits 4rt- the rudder angle 6 to the processor 
1. The rudder angle 8 here indicates at what angle the steering 
wheel is turned with respect to the initial position . The steering 
5 wheel is considered to be in the initial position when the steering 
wheel is not turned, that is, when the vehicle Vusr is in the 
straight-ahead position. The display device 6 is typically a 
liquid crystal display. 

[0019] Described next is the operation of ouch the drive 
10 assistant device Uastl . When the driver wants assistance by — from 
the drive assistant device Vastl, the processor 1 starts executing 
the program PGa . 

[0020] Refer now to a flowchart e#- in FIG . 5 for the processing 
procedure in the processor 1 written in the program PGa. In FIG. 

15 5, the processor 1 first generates an image capture instruction 
Icpt, and transmits j^fe— the image capture instruction Icpt to the 
image capture device 4 (step SI) . Here, as shown in FIG. 5, the 
procedure returns to step SI after step S10 is through completed , 
and the processor 1 generates another image capture instruction 

20 Icpt. The program PGa is ^e-written so that a time interval between 
those two image capture instructions Icpt is substantially a tl 
second. Here, the value of tl is -&e-selected so as to allow the 
display device 6 to display the display image Sout for 30 frames 
per second. Herein, the image capture instruction Icpt is a signal 

25 instructing the image capture device 4 for image capturing. The 
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image capture device 4 responsively captures such a captured image 
Sept such as shown in FIG. 4, and stores -jrfe— the captured image 
Sept in frame memory (not shown) reserved in the working area 3 
(step S2) . 

5 [0021] The processor 1 then watches a deriving timing Tl (step 
S3) . This deriving timing Tl is previously written in the program 
PGa, and allows the processor 1 to derive the left- and right-side 
trajectories Ppl and Pp2 once every t2 second. The value of t2 
is selected to be larger than that of tl (e.g., 0.1 second) since 

10 a change on a time base in the rudder angle 6 is small. 

[0022] In the deriving timing Tl, the processor 1 generates 
a detection instruction Idtc, and transmits the detection 
instruction Idtc 4Jz-to the rudder angle sensor 5 (step S4) . The 
detection instruction Idtc is a signal instructing the rudder angle 

15 sensor 5 to detect the rudder angle 9 . The rudder angle sensor 
5 responsively detects the rudder angle 6 , and stores jrfc- the 
rudder angle 8 in the working area 3 (step S5) . 
[0023] Based on thus the detected rudder angle 6 , the 
processor 1 derives the left- and right-side trajectories Ppl and 

20 Pp2 (step S6) . More specifically, derived by the processor 1 here 
are equations respectively for the left- and right-side 
trajectories Ppl and Pp2 under the Ackermann's model. Here, in 
the strict sense, the left- and right-side trajectories Ppl and 
Pp2 are defined as being trajectories that are traced by left- 

25 and right-rear wheels of the vehicle Vusr on the condition that 
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the driver keeps the steering wheel at the currently derived rudder 
angle 8 . The left-side trajectory Ppl that is calculated by such 
an equation becomes an arc in a predetermined length. In more 
detail, the arc is a segment of a circle which is traceable by 
5 the vehicle Vusr Vusr around a circling center of the circle . 
The radius of the circle is equal to a distance from the circling 
center of the circle to a point having a rotation center of the 
left-rear wheel projected onto the road surface. The equation 
for the right-side trajectory Pp2 is similar except that the arc 
10 is traced by the right-rear wheel, on its rotation center, of the 
vehicle Vusr. 

[0024] Then, the processor 1 generates overlaying position data 
Dsp indicating where to overlay the two indicators Sincil and Sind2, 
and stores the data Dsp in the working area 3 (step S7) . As an 

15 example, if derived in step S 6 arc ouch the left- and right-side 
trajectories Ppl and Pp2 as shown in FIG. 6 are derived in step 
S6 , the processor 1 calculates two points aO and bO being which 
are closest to the vehicle Vusr (not shown) on those trajectories 
Ppl and Pp2, respectively. The processor 1 then calculates a point 

20 al being away by which is a predetermined distance Ad away from 
the point aO on the left-side trajectory Ppl, and a point hi being 
away which is also hy — the predetermined distance Ad away from 
the point bO on the right-side trajectory Pp2 . The processor 1 
repeats the same processing until i (where i is a natural number 

25 being 2 or larger) sets of coordinates such as {aO, bO) , [al , bl) , . . . , 
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(a(i-l)f b{i-l)) are calculated. The sets of coordinates are 
numbered starting from the one closest to the vehicle Vusr. 
Accordingly, as shown in FIG. 7 , stored in the working area 3 is 
the overlaying position data Dsp including those numbered sets 
5 of coordinates are stored in the working area 3 . 

[0025] Based on the overlaying position data Dsp and the 
aforementioned captured image Sept, the processor 1 then generates 
a frame of the display image Sout on the frame memory (step S8) . 
Here, as already described by referring with reference to FIG. 

10 2, the display image Sout is the one having the indicators Sindl 
and Sind2 overlaid on the captured image Sept. In step S8, more 
4^n— in more detail, the processor 1 first selects, from the 
overlaying position data Dsp generated in step S7, a set of 
coordinates which is not yet selected and which are the smallest 

15 in number. In this example, since **e- a set has not yet been 
selected, Qclectcd now io the set of [aO, bO) is now selected . 
The processor 1 then overlays the indicators Sindl and Sind2 onto 
the points aO and bO in the captured image Sept on the frame memory. 
After this overlaying process, such a display image Sout as the 

20 one shown in FIG. 8 is generated for one frame on the frame memory. 
[0026] The processor 1 then transfers the display image Sout 

on the frame memory to the display device 6 for display to be 
displayed thereon (step S9) . In the current display image Sout 
on the display device 6, the indicator Sindl is overlaid on the 

25 point aO on the left-side trajectory Ppl, and the indicator Sind2 




is overlaid on the point bO on the right-side trajectory Pp2 . 
[0027] Then, the processor 1 determines whether now it is now 
the time to end the processing of FIG. 5 (step S10) . If determined 
not yet the processor 1 determines that the processing should not 
5 end , the procedure returns to step SI for generating another display 
image Sout . By the time when steps SI and S2 are through completed , 
another captured image Sept is newly stored on the frame memory. 
Then^ in step S3, if the processor determines determining that 
the timing Tl has not come yet, the processor 1 then watches a 

10 timing T2 to change the overlaying positions of the indicators 
Sindl and Sind2 {step Sll) . Here, the changing timing T2 is 
previously written in the program PGa, and allows the processor 
1 to change the overlaying positions of the indicators Sindl and 
Sind2 once every t3 second. If the value of T3 is set too small, 

15 the indicator Sindl moves too fast from the point aO to al for 
the driver to follow with her/his eyes on the display device 6. 
With consideration therefor, the value of t3 is selected to be 
larger than that of tl {e.g., 0.05 second). 

[0028] If the processor 1 determines that the timing T2 has 
20 not come yet, the processor 1 generates a frame of the display 
image Sout on the frame memory {step S12) . This is based on the 
captured image Sept stored in step S2 and the set of coordinates 
currently selected in the overlaying position data Dsp {in this 
example, the set of [a0, bO) ) . As such, the resulting display 
25 image Sout is also the one having the indicators Sindl and Sind2 




overlaid on the points aO and bO on the captured image Sept. Then, 
the processor 1 transfers thuo the generated display image Sout 
on the frame memory to the display device 6 for display to be 
displayed thereon (step S13) . 
5 [0029] Next, in step S10, if the processor 1 determines that 
now it is now not the time to end the processing of FIG. 5, the 
procedure returns to step SI. By the time when steps SI and S2 
are through completed , another captured image Sept is newly stored 
on the frame memory . Then_^ in step S3, if the processor 1 determines 

10 that the timing Tl has not come yet, and in step Sll , if the processor 
1 determines that the timing T2 is now right, the procedure goes 
to step S14. Then, the processor 1 selects, from the overlaying 
position data Dsp on the working area 3, a set of coordinates which 
is not yet selected and which are the smallest in number (step 

15 S14). Since the set which was last selected last is [aO, bO) , 
selected this time io the set (al, bl) is now selected . 
[0030] Next, the processor 1 generates a new frame of the display 
image Sout on the frame memory based on the captured image Sept 
and the set of coordinate coordinates (in this example, the set 

20 of {al, bl) ) currently selected in the overlaying position data 
Dsp (step S15) . As such, as shown in FIG. 9, the resulting display 
image Sout is the one having the indicators Sindl and Sind2 overlaid 
on the points al and bl on the captured image Sept. Then, the 
processor 1 transfers thuo the generated display image Sout on 

25 the frame memory to the display device 6 for display to be displayed 
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thereon (step S16) . 

[0031] Such steps SI to S16 are repeated until the determination 
in step S10 becomes Yes to end the processing of FIG. 5. In this 
manner, the overlaying positions of the indicators Sindl and Sind2 
5 change, in increments of the predetermined distance Ad, from the 
points aO and bO to a(i-l) and b(i-l), respectively. Thus, the 
indicators Sindl and Sind2 are displayed as if moving in the same 
heading direction as the vehicle Vusr is heading towards along 
the left- and right-side trajectories Ppl and Pp2 . What is good 

10 Advantageously , as the those indicators Sindl and Sind2 are 
displayed on an intermittent basis, the left- and right-side 
trajectories Ppl and Pp2 are also displayed on an intermittent 
basis on the display device 6. Accordingly, the left- and 
right-side trajectories Ppl and Pp2 become more noticeable and 

15 are emphasized to a further degree. With such indicators Sindl 
and Sind2 t the driver can instantaneously locate the trajectories 
Ppl and Pp2 in the display image Sout. 

[0032] Further, every time the rudder angle 6 comes from is 
detected by the rudder angle sensor 5 according to the deriving 

20 timing Tl, the processor 1 derives the left- and right-trajectories 
Ppl and Pp2 based on the current rudder angle 6 . In this manner, 
the trajectories Ppl and Pp2 displayed on the display device 6 
always become alwayo responsive to the driver's steering. 
[0033] Note that, in the first embodiment, the changing timing 

25 T2 may be variable. For example, in the case that where the 
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overlaying positions of the indicators Sindl and Sind2 are 
relatively close to the vehicle Vusr, the program PGa may be *e 
written so that the changing timing T2 comes earlier. If so, the 
left- and right-side trajectories Ppl and Pp2 become easier to 
5 notice. 

[0034] Further, in the first embodiment, the predetermined 
distance Ad between two successive points of aj and a{j + l) is 
constant on the left-side trajectory Ppl . Here, the value j is 
a positive integer between 0 and . The predetermined distance 

10 Ad may not necessarily be constant. For example, in the case 
that where the point aj is relatively close to the vehicle Vusr, 
the program PGa may be ee-written so that the predetermined distance 
Ad is set to be relatively small so as to cause the processor 
1 to select the point a(j+l). Conversely, the program PGa may 

15 be se-written so that the predetermined distance Ad is set to 
be relatively large so as to cause the processor 1 to select the 
pointa{j + l) . Inboth cases, the left- and right-side trajectories 
Ppl and Pp2 become conspicuous to a further degree. 
[0035] FIG. 10 is a block diagram showing the hardware structure 

2 0 of a rendering device Urnd2 according to a second embodiment of 
the present invention. In FIG. 10, the rendering device Urnd2 
includes a processor 21, ajprog ram memory 22, and a working area 
23. The program memory 22 is typified by ROM (Read Only Memory) , 
and stores a program PGb for defining the processing procedure 

25 in the processor 21. By following the program PGb f the processor 



it 




21 generates such a display image Sout such as the one shown in 
FIG. 11. The display image Sout shows an estimated path Pp of 
the vehicle Vusr (see FIG. 3) to be traced by a left-rear wheel 
of the vehicle Vusr. The estimated path Pp is displayed only 
5 during a display time period Pdt, which will be later described 
described later . 

[0036] The working area 3 is typified by RAM (Random Access 
Memory) , and is used when the processor 21 executes the program 
PGb. The rendering device Urnd2 irn— according to the above 

10 above-described structure is typically incorporated in a drive 
assistant device Uast2 . Here, as to the drive assistant device 
Uast2 f the only structural difference from the drive assistant 
device Uastl of the first embodiment is including that the drive 
assistance Uast2 includes the rendering device Urnd2 instead of 

15 the rendering device Urndl. Thus, any component appeared 
illustrated in FIG. 1 io under has the same reference numeral 
in FIG. 10, and therefore is not described again. 
[0037] Described next is the operation of ouch the drive 
assistant device Uast2 . When the driver wants assistance fey— from 

20 the drive assistant device Uast2 f the processor 21 starts executing 
the program PGb in the program memory 22. 

[0038] Refer now to a flowchart in FIG. 12 for the processing 
procedure in the processor 21 written in the program PGb. Compared 
with FIG. 5, the flowchart of FIG. 12 includes the same steps, 
25 and thus those are under steps having the same step numbers are 
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identical and thus are not described again. 

[0039] First, by going through steps SI to S6, the processor 
21 derives an equation for the estimated path Pp. The procedure 
then goes to step S21, and the processor 21 generates the display 
5 image Sout based on the captured image Sept stored in step S2 and 
the estimated path Pp derived in step S6. More specifically, the 
processor 21 renders thus the derived estimated path Pp in its 
entirety on the display image Sout, and the resulting display image 
Sout looks a-s — like the one shown in FIG. 11. 

10 [0040] The procedure then goes to step S9, and the processor 
21 transfers the display image Sout currently on the frame memory 
to the display device 6 for display to be displayed thereon . Then, 
the processor 21 determines whether now it is now the time to 
end the processing of FIG. 12 (step S 10) , and if not yot the processor 

15 I determines that the processing should not end now , the procedure 
returns to step SI for generating another display image Sout on 
the frame memory. By the time when steps SI and S2 are through 
completed , another captured image Sept is newly stored on the frame 
memory. Then^ in step S3, if the processor 1 determines that the 

20 timing Tl has not come yet, the processor 1 then determines whether 
it is now is-in the display time period Pdt for the estimated path 
Pp (step S22) . Here, the display time period Pdf is previously 
written in the program PGb, and comes every t4 second in this the 
second embodiment. This m eans that the estimated path Pp 

25 appears on and disappears from the display with a time lapse of 
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t4 second. Note that, if the value of t4 is set too small, the 
appearance and disappearance of the estimated path Pp will be too 
swift for the driver to notice. With consideration therefor, the 
value of t4 is selected to be larger than that of tl (e.g., 0.1 
5 second) . 

[0041] If the processor 21 determines that it is now -arS-in the 
display time period Pdt, the procedure goes to step S21. The 
processor 21 then generates, on the frame memory, the display image 
Sout including the estimated path Pp (see FIG. 11) . The procedure 

10 then goes to step S9, and the processor 21 transfers the current 
display image Sout on the frame memory to the display device 6 
for display to be displayed thereon. Then, the processor 21 
determines whether it is now 4rS- the time to end the processing 
of FIG. 12 (step S10) , and if not yet the processor 1 determines 

15 that the processing should not end now , the procedure returns to 
step SI for generating another display image Sout. In step S3, 
if the processor 21 determines that the deriving timing Tl has 
not come yet, and in step S22, if the processor 1 determines that 
now the present time is not in the display time period Pdt, the 

20 procedure goes to step S23 . In step 23 , the processor 2 1 transfers , 
to the display device 6 for display to be displayed thereon , the 
captured image Sept stored in step S2 (see FIG. 4) as the display 
image Sout without any change changes thereto (step S23) . 
[0042] Such steps SI to S23 are repeated until the determination 

25 in step S10 becomes Yes to end the processing of FIG. 12. In this 




manner, the estimated path Pp is displayed only during the display 
time period Pdt. The estimated path Pp appears on and disappears 
from the display on an intermittent basis. Accordingly, the 
estimated path Pp becomes noticeable, and the driver finds it easy 
5 to locate the estimated path Pp in the display image Sout. 

[0043] FIG. 13 is a block diagram showing the hardware structure 
of a rendering device Urnd3 according to a third embodiment of 
the present invention. In FIG. 13, the rendering device Urnd3 
includes a processor 41, a_program memory 42, and a working area 

10 43. The program memory 42 is typified by ROM (Read Only Memory) , 
and stores a program PGc for defining the processing procedure 
in the processor 41. By following the program PGc, the processor 
41 generates ouch a display image Sout such as the one shown in 
FIG. 14. The display image Sout shows an estimated region Rpt 

15 on a road surface Frd for the vehicle Vusr (see FIG. 3) to move. 
Specifically, the estimated region Rpt is defined by the left- 
and right-side trajectories Ppl and Pp2 described above in the 
first embodiment, and a line segment Llmt passing through a no-go 
point Plmt. Here, the no-go point Plmt is a point indicating the 

20 farthest limit for the vehicle Vusr to move, and if the vehicle 
Vusr keeps moving, irfe — the vehicle might first collide into the 
obstacle Vbst fir ot . 

[0044] The working area 43 is typified by RAM {Random Access 
Memory), and is used when the processor 41 executes the program 
25 PGc. The rendering device Urnd3 irfl— according to the above 

20 




above-described structure is typically incorporated in a drive 
assistant device Uast3. Here, as to the drive assistant device 
Uast3, the structural difference from between the drive assistant 
device Uastl and the drive assistant device Uast3 is that the drive 
5 assistant device Uast3 includes io including the rendering device 
Urnd3 instead of the rendering device Urndl, and further including 
4- includes four active sensors 441 to 444, which is exemplified 
for herein as a measuring sensor in Claims . These are the only 
structural differences, and thus any component appeared 

10 illustrated in FIG. 1 io under has the same reference numeral 
in FIG. 13, and therefore is not described again. 
[0045] As shown in FIG. 15A, the active sensors 441 to 444 are 
embedded in the rear-end of the vehicle Vusr, preferably, in a 
lateral direction. The active sensors 441 to 444 thus arranged 

15 as ouch emit ultrasonic waves or radio waves toward the area to 
the rear of the vehicle Vusr, andmonitor reflected waves . Thereby, 
as shown in FIG. 15B, distances dl to d4 to an obstacle Vbst located 
closest behind the vehicle Vusr are detected by the active sensors 
441 to 444. 

20 [0046] Described next is the operation of ouch the drive 
assistant device Uast3. When the driver wants assistance ky- from 
the drive assistant device Uast3, the processor 41 starts executing 
the program PGc in the program memory 42. 

[0047]. Refer now to a flowchart e# - in FIG. 1 6 for the processing 
25 procedure in the processor 41 written in the program PGc, In FIG. 
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16, the processor 41 first generates a distance measuring 



instruction Imsr to all of the active sensors 441 to 444 (step 
S41) . Here, the distance measuring instruction Imsr is a signal 
5 to instruct all of the active sensors 441 to 444 to detect the 
distances dl to d4, and to_transmit those distances to the processor 
41. The active sensors 441 to 444 each responsively perform such 
detection, and store the resultant distances dl to d4 to the working 
area 43 (step S42) . 

10 [0048] Next, based on the thuo detected distances dl to d4 , 
the processor 41 calculates coordinates {xl, yl) to {x4, y4) of 
four points PI to P4 on the surface of the object Vbst (step S43) . 
Referring to FIG . 17 , the process in step S43 is described in detail . 
FIG. 17 shows the vehicle Vusr, the obstacle Vbst, and a 

15 two-dimensional (2D) coordinate system. In the 2D coordinate 
system, the Y-axis connects a rotation center of a left-rear wheel 
Wrl and that of a right-rear wheel Wr2 . With respect to the Y-axis , 
the X-axis is a perpendicular bisector parallel to a horizontal 
plane. As described above, the active sensors 441 to 444 are 

20 securely placed in the vehicle Vusr. Therefore, positions Al to 
A4 of the active sensors 441 to 444 from which the ultrasonic waves, 
for example, are emitted can be all defined by coordinates (xal f 
yal) to {xa4, ya4) that are known in the 2D coordinate system. 
Also, angles 0 1 to 0 4 at which the active sensors 441 to 444 emit 

25 the ultrasonic waves are known. In this the third embodiment, 



instruction Imsr, and transmits 




measuring 
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the angles <$> 1 to <f> 4 are formed by the X-axis and the emitted waves, 
and FIG . 17 exemplarily shows only the angle <f> 1 . As such, the 
above coordinates (xl, yl) is equal to (dl • cos<t> 1+xal, dl * sin 
<t>l+yal), and those coordinates (x2, y2) to (x4 to y4) are equal 
5 to (d2 • cos $ 2+xa2 f d2 • sin $ 2+ya2) to (d4 • cos <t> 4+xa4 , d4 • sin 
<M+ya4), respectively. 

[0049] Then, based on the thuo calculated four points PI to 
P4, the processor 41 calculates coordinates (xlmt, ylmt) of the 
corner point Pcnr of the obstacle Vbst as one example of the no-go 
10 point Plmt {step S44) . By referring to FIG. 18, the process in 
FIG . — 44— step S44 is now described in detail. The processor 41 
first performs a Hough transform with respect to the points PI 
to P4 so that curves CI to C4 are derived in the Hough space which 
is defined by the p -axis and 0-axis. Here, the curves CI to C4 
15 are expressed as the following equations (1) to (4), respectively. 
p = xl • cos 9 +yl • sin 9 ...(1) 
p= x2 • cos0+y2-sin 0 ...(2) 
p= x3 * cos 6 +y3 • sin 9 ...(3) 
p = x4 * cos 8 +y4 * sin 6 ...(4) 
20 [0050] According to the above equations (1) and (2), the 
processor 41 calculates coordinates (pi, 6 1) of an intersection 
point Pel of the curves CI and C2 in the Hough space, and according 
to the equations (2 ) to (4 ) , theprocessor 41 calculates coordinates 
(p2, 9 2) of an intersection point Pc2 of the curves C2 to C4 
25 in the Hough space . From the intersection point Pel , the processor 
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41 then derives an equation for a straight line Pi P2 . Here, the 
line PI P2 is expressed by the following equation (5) on the 2D 
coordinate system. Similarly, a line P2 P4 is expressed by an 
the following equation (6) . 
5 y= {-cos 6 1 • x+ p 1) /sin 0 1 ...(5) 

y= (-cos0 2 ■ x+/o2)/sine2 -..(6) 

From those equations (5) and (6), the processor 41 
calculates coordinates of an intersection point of the line Pi 
P2 and the line P2 P3, and the resulting coordinates are determined 
10 as the above-mentioned coordinates {xlmt, ylmt) . 

[0051] By similarly going through steps S4 and S5 of FIG. 5, 
the processor 41 receives the current rudder angle 6 of the vehicle 
Vusr (steps S45 and S46) . 

[0052] The processor 41 then calculates, in the 2D coordinate 
15 system, coordinates {xcnt, ycnt) of a center point Pent (see FIG. 
19) of the circle traceable by the vehicle V'usr when rotated (step 
S47). The processor 41 also derives equations for circles Crl 
and Cr2, which are traced respectively by the left- and right-rear 
wheels Wrl and Wr2 f on each rotation center, of the vehicle Vusr 
20 when rotated around the center point Pent (stepS48). Here, since 
the coordinates {xent, ycnt)- and the equations for the circles 
Crl and Cr2 are easily calculated under the well-known Ackermann' s 
model, steps S47 and S48 are not described in detail. Further, 
the circles Crl and Cr2 include the left- and right-side 
25 trajectories Ppl and Pp2 described above in the first embodiment. 




[0053] The processor 41 then derives an equation for a straight 
line Llmt, which passes through the coordinates {xcnr, ycnr) 
calculated in step S44, and the coordinates (xcnt, ycnt) calculated 
in step S47 (step S49) . Herein, the straight line Llmt specifies 
5 the farthest limit for the vehicle Vusr to move without colliding 
with the obstacle Vbst. 

[0054] The processor 41 next generates the estimated region 
Rpt, which is a region that is enclosed by the circles Crl and 
Cr2 calculated in step S48, the straight line Llmt calculated in 

10 step S49, and a line segment Lrl2 (step S410) . Here, the line 
segment Lrl2 is the one connecting which connects the rotation 
centers of the left- and right-rear wheels Wrl and Wr2. 
[0055] By similarly going through steps SI and S2 of FIG. 5, 
the processor 41 receives the captured image Sept from the image 

15 capture device 4 (steps S411, S412) . Based on the captured image 
Sept and the estimated region Rpt generated in step S410, the 
processor 41 then generates the display image Sout on the frame 
memory. More specifically, the processor 41 deforms the estimated 
region Rpt to the one viewed from the image capture device 4, and 

20 renders that the estimated region Rpf on the captured image Sept. 
The resulting display image Sout looks a-s— like the one shown in 
FIG. 14. The processor 41 then transfers the display image Sout 
on the frame memory to the display device 6 for dioplay to be 
displayed thereon (step S414) . Such steps S41 to S414 are repeated 

25 until the determination becomes Yes in step S415 to end the 
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processing of FIG. 16. As such, as the estimated region Rpt extends 
to the no-go point Plmt, the driver can instantaneously know the 
farthest limit to move the vehicle Vusr. 

[0056] In the above first to third embodiments as described 
5 above , the image capture device 4 is embedded in the rear-end of 
the vehicle Vusr. This is not restrictive The present invention, 
however, is not restricted thereto , and the image capture device 
4 can be embedded in the front-end of the vehicle Vus r will also 
de. Further, the number of image capture devices 4 is not limited 

10 to one, andmay be more than one depending on the design requirements 
of the drive assistant devices Uastl to Uast3. 
[0057] Still further, in the above — above-described 
embodiments, the captured image Sept is the one on which the left- 
and right-side trajectories Ppl and Pp2, the estimated path Pp, 

15 and the estimated region Rpt are rendered. Here, the captured 
image Sept may be subjected to some image processing by the 
processors 1, 21, and 41 before having those rendered thereon. 
Such image processing is typified by processing of generating an 
image of an area around the vehicle Vusr viewed from a virtual 

20 viewpoint set high up in the vehicle Vusr. 

[0058] Still further, in the above first to third embodiments 
described above , the captured image Sept is stored in the frame 
memory in response to the image capture instruction Icpt 
transmitted from the processors 1, 21, and 41 to the image capture 

25 device 4 . Thio io not rootrictivc The present invention, however, 
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is not restricted thereto , and the captured image Sept is 
voluntarily generated by the image capture device A-r 4 and then 
stored in the frame memory. Similarly, the rudder angle 6 may 
be detected voluntarily by the rudder angle sensor 5 without 
5 responding to the detection instruction Idct coming originating 
from the processors 1, 21, and 41. 

[0059] Still further, in the abovo third embodiment described 
above , four active sensors 441 to 444 are placed in the vehicle 
Vusr. The number thereof io not restrictive present invention, 

10 however, is not restricted thereto , and may bo one or more active 
sensors may be placed in the vehicle Vusr . Here, if only one active 
sensor is placed in the vehicle Vusr , the direction of the lens 
thereof needs to be dynamically changed so that the angle <f> of 
the emitted waves is set to be wider. 

15 [0060] Still further, in the above above-described third 
embodiment, the active sensors 441 to 444 are provided herein as 
one example of a measuring sensor in Claims for measuring the 
distances dl to d4 to the obstacle Vbst. Thia ia not rcotrictive 
The present invention, however, is not restricted thereto , and 

20 other another type of measuring sensor such as a passive sensor 
may be used. Here, to structure such an exemplary passive sensor, 
two image capture devices are required to cover the area to the 
rear of the vehicle Vusr. These image capture devices each pick 
up an image of the obstacle Vbst located behind the vehicle Vusr. 

25 Based on a parallax of the obstacle in images, the processor 41 




then measures a distance to the obstacle Vbst with stereoscopic 
views (stereoscopic vision) . 

[0061] Still further, in the above — above-described 
embodiments, the programs PGa to PGc are stored in the rendering 
5 devices Urndl to Urnd3, respectively. Thio is not restrictive 
The present invention, however, is not restricted thereto , and 
those programs PGa to PGc may be distributed in a recording medium 
typified by a CD-ROM, or over a communications network such as 
the Internet. 

10 [0062] While the present invention has been described in detail, 
the foregoing description is in all aspects illustrative and not 
restrictive. It is to be understood that numerous other 
modifications and variations can be devised without departing from 
the scope of the invention. 
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ABSTRACT OF THE DISCLOSURE 

In a rendering device Urndl , a processor -3r-derives an 
estimated path to be traced by a- the left- and right-rear wheels 
5 of a vehicle based on a rudder angle that is provided by a 
rudder angle sensor-^. The processor i then determines positions 
for overlaying indicators on the derived estimated path. The 
processor -3^-then renders the indicators on thug the determined 
points in a captured image which is provided by an image capture 

10 device, and generates a display image. Hero, in In the display 
image, the indicators move along the estimated path in the heading 
direction e^-the vehicle is heading towards . In this manner, the 
estimated path in the display image that is generated by the 
rendering device Urndl becomes noticeable for a driver of the 

15 vehicle. 
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